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LONG-TERM  GOAL 

The  long-term  goal  of  this  research  is  to  construct  the  Navy’s  next-generation  global  numerical 
weather  prediction  (NWP)  model  using  new  numerical  methods  specifically  designed  for  distributed- 
memory  computers.  To  take  full  advantage  of  the  new  computer  architectures,  the  spherical  global 
domain  must  be  partitioned  into  local  sub-domains,  or  elements,  which  can  then  be  solved 
independently  on  the  multiple  processors  of  these  computers.  The  numerical  methods  used  on  these 
sub-domains  must  be  not  only  local  in  nature  but  also  high-order  in  accuracy  and  highly  efficient. 

Thus  the  final  objective  of  this  project  is  to  construct  a  new  global  NWP  model  which  is  as  accurate  as 
the  current  spectral  model  Navy  Operational  Global  Atmospheric  Prediction  Systems  (NOGAPS) 
while  much  more  efficient,  thereby  allowing  for  finer  resolution  forecasts. 

OBJECTIVES 

The  objective  of  this  project  is  to  construct  high-order  local  methods  for  the  Navy’s  next-generation 
global  NWP  model.  The  high-order  accuracy  of  these  methods  will  ensure  that  the  new  model  yields 
the  same  accuracy  as  the  current  spectral  model  while  the  locality  of  these  methods  will  ensure  that  the 
efficiency  of  the  model  increases. 

APPROACH 

To  meet  our  objectives  we  explore: 

1 .  spectral  element  (SE)  methods, 

2.  spectral  elements  in  space  with  semi-implicit  and  semi-Lagrangian  methods  in  time,  (SESL) 

3.  vertical  mode  decomposition  to  reduce  the  Helmholtz  equation  from  3  to  2  dimensions, 

4.  fast  matrix  solvers  on  distributed-memory  computers. 

The  power  of  spectral  element  methods  is  that  they  are  high-order  accurate,  like  spectral  transform 
methods  (i.e.,  spherical  harmonics),  yet  are  completely  local  in  nature  -  meaning  that  the  equations  are 
solved  independently  within  each  individual  element  and  processor.  Semi-Lagrangian  methods  are 
also  being  considered  because  these  methods,  like  high-order  methods,  have  minimal  dispersion  errors. 
This  property  is  important  for  properly  capturing  fine-scale  atmospheric  phenomena  (e.g.,  tropical 
cyclones).  In  addition,  semi-implicit  and  semi-Lagrangian  methods  offer  vast  improvements  in 
efficiency  due  to  the  longer  time  steps  that  they  pennit. 

After  validation  of  the  spectral  element,  semi-implicit,  and  semi-Lagrangian  discretization  on 
barotropic  test  cases,  the  vertical  integration  scheme  for  the  full  3D  primitive  equations  needs  to  be 
scrutinized.  The  semi-implicit  time-integration  scheme  of  the  3D  primitive  equation  model,  which  we 
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call  the  NRL  spectral  element  atmospheric  model  (NSEAM),  must  then  be  ported  to  the  message¬ 
passing  interface  (MPI).  At  this  point  direct  comparisons  with  the  current  version  of  NOGAPS  must 
be  made.  Superior  performance  of  NSEAM  over  NOGAPS  would  then  justify  the  further  effort  to 
make  NSEAM  the  operational  next-generation  NWP  model.  To  accomplish  the  final  objective,  the 
NSEAM  dynamical  core  needs  to  be  coupled  with  the  physical  parameterization  package  already  in 
existence  within  NOGAPS. 

WORK  COMPLETED 

The  SE  and  SESL  discretization  methods  were  tested  on  the  spherical  shallow  water  equations.  These 
methods  have  been  tested  extensively  and  two  papers  have  been  written  concerning  these  innovative 
approaches  (Ref.  [1,2]).  The  3D  model,  NSEAM,  has  undergone  extensive  validation  and  performance 
testing.  Four  test  cases  have  been  applied  to  NSEAM  and  this  work  was  presented  at  a  conference  in 
Garching  Gennany  (Ref.  [4])  and  at  an  invited  talk  at  the  Max-Planck  Institute  in  Hamburg  Gennany 
(March  2003).  The  parallel  version  of  the  semi-implicit  time-discretization  of  NSEAM  has  been 
completed.  The  comparison  of  NSEAM  to  NOGAPS  can  be  summarized  as  follows:  NSEAM  yields 
the  same  exponential  accuracy  as  NOGAPS  while  being  at  least  50%  faster  (at  T239  L30  resolution). 
At  the  next  projected  operational  resolution  of  T479  L60,  a  conservative  estimate  shows  that  NSEAM 
will  be  anywhere  from  100%  to  200%  faster  than  NOGAPS.  The  impressive  perfonnance  of  NSEAM 
is  due  to  the  fast  GMRES  iterative  solver  used  to  solve  the  Helmholtz  operator  resulting  from  the  semi- 
implicit  time-discretization.  In  addition,  instead  of  solving  a  full  3D  Helmholtz  operator,  the  equations 
are  decoupled  into  a  series  of  2D  Helmholtz  operators  via  vertical  mode  decomposition.  Since  only 
the  fastest  three  modes  need  to  be  solved,  then  the  cost  of  the  semi-implicit  is  only  three  2D  Helmholtz 
operators,  which  is  equivalent  to  solving  three  shallow  water  models. 

RESULTS 

Horizontal  Operators.  To  show  the  accuracy  of  the  SE  method  that  is  used  to  discretize  the  horizontal 
operators  in  NSEAM,  in  Fig.  1  we  show  results  for  nonlinear  zonal  geostrophic  flow  (barotropic  Case 
2  in  Ref.  [3]).  Note  the  high-order  spectral  accuracy  achieved  by  NSEAM. 
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Figure  1:  L2  error  as  a  function  of  horizontal  resolution  for  the  NRL 's  spectral  element 
atmospheric  model  (NSEAM),  Japanese  earth  simulator  finite  volume,  CSU  finite  volume  code,  and 

the  NCAR  spherical  harmonics  model. 


NSEAM  is  more  accurate  than  the  other  three  models:  two  finite  volume  models  and  a  spectral 
transfonn  model.  In  Ref.  [3]  we  show  that  NSEAM  is  indeed  more  accurate  than  all  currently  existing 
models.  A  similar  result  is  obtained  with  SESL  (Ref.  [2])  while  using  time-steps  10  times  larger. 

3D  Model.  As  a  first  test  to  validate  the  3D,  model  a  Rossby-Haurwitz  wave  number  4  was  run  both 
with  NSEAM  and  NOGAPS.  This  test  simulated  a  5-day  forecast  using  a  resolution  of  T80L24  and 
the  results  show  that  both  models  yield  identical  results  (see  Ref.  [3]  and  ONR  2002  report). 
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Figure  2:  Zonally-averaged  zonal  velocity  (U)  for  the  NRL  spectral  element  atmospheric  model 
(NSEAM)  running  the  Held-Suarez  test  case  with  T64  L20  as  a  function  of  latitude  and  vertical 

coordinate  for  a  1200-day  integration. 

As  a  second  test,  NSEAM  was  run  with  the  Held-Suarez  test  case.  This  case  is  the  primary  test  for  all 
new  atmospheric  models  because  it  mimics  a  realistic  atmosphere  with  physical  parameterization. 
Figure  2  shows  the  contour  plot  of  the  mean  zonally  averaged  zonal  velocity  as  a  function  of  latitude 
and  vertical  coordinate  for  T64  L20  for  a  1200-day  integration.  Figure  2  compares  well  with  the 
results  of  Held-Suarez  and  those  from  the  European  Center  for  Medium-Range  Weather  Forecasting 
(ECMWF)  spectral  transform  model  and  clearly  shows  the  fonnation  of  the  jet  stream  (in  the  mid¬ 
latitudes).  In  addition,  the  model  is  stable  for  long  time-integrations. 

As  a  third  test,  baroclinic  instability  was  studied.  This  case  is  a  two-part  test:  in  the  first  part,  a  30-day 
simulation  is  executed  to  ensure  that  the  model  maintains  the  initially  balanced  state.  The  second  part 
involves  adding  a  perturbation  to  the  initial  state  and  then  tracing  the  evolution  of  the  baroclinic 
instability.  NSEAM  was  able  to  maintain  this  balanced  state  for  30  days  and  beyond  thereby 
confirming  that  the  model  generates  no  spurious  waves.  This  test  is  of  interest  because  other  geodesic 
grid  models  have  yet  to  remain  stable  due  to  unphysical  number  5  waves  generated  by  the  grid  in 
conjunction  with  the  low-order  methods.  This  test  clearly  shows  the  need  for  high-order  numerics  in 
atmospheric  models  when  unstructured  irregular  grids  are  used.  In  Fig.  3  we  plot  the  minimum  surface 
pressure  for  the  baroclinic  instability  for  NSEAM  and  three  well-established  models.  They  are:  the 
icosahedral  model  of  the  German  Weather  Service  (GME),  the  Lin-Rood  NASA  flux-form  semi- 
Lagrangian  model  (NASA),  and  the  National  Center  for  Atmospheric  Research  (NCAR)  spectral 
transfonn  model.  The  two  dotted  lines  (GME  and  NASA)  are  both  low-order  models  while  the  two 
solid  lines  (NSEAM  and  NCAR)  represent  high-order  models.  Note  that  the  two  low-order  models 
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behave  similarly  while  the  two  high-order  behave  similarly  to  each  other.  This  test  shows  that  while 
NSEAM  is  a  grid  point  model  it  does  behave  like  a  spectral  transform  model  and  this  behaviour  is 
directly  due  to  the  high-order  numerics  that  it  uses. 


Figure3:  The  minimum  surface  pressure  as  a  function  of  days  for  the  NRL ’s  spectral  element 
atmospheric  model  (NSEAM),  the  German  Weather  Service  icosahedral  model  (GME),  the  NASA 
flux-form  semi-Lagrangian  model  (NASA),  and  the  NCAR  spectral  transform  model  (NCAR). 

Finally,  to  show  the  scalability  of  the  Navy’s  next-generation  model,  in  Fig.  4  we  compare  NSEAM 
with  NOGAPS  using  the  maximum  time-step  that  each  model  can  use  with  the  operational  resolution 
T239  L30.  No  physical  parameterization  is  used  in  this  comparison;  only  the  dynamical  cores  of  the 
models  are  used. 

This  result  clearly  shows  that  NSEAM  can  use  far  more  processors  than  NOGAPS  and  it  also  scales 
better  than  NOGAPS.  At  this  spatial  resolution  NSEAM  can  easily  use  20,000  processors  while 
NOGAPS  can  only  accommodate  240  processors  efficiently.  Very  few  models  will  be  able  to  scale  at 
the  rate  of  NSEAM. 

IMPACT 

NOGAPS  is  run  operationally  by  the  Fleet  Numerical  Meterology  and  Oceanography  Center 
(FNMOC)  and  is  the  heart  of  the  Navy’s  operational  support  to  nearly  all  DOD  users  worldwide.  This 
work  targets  the  next-generation  of  this  system  for  massively  parallel  computer  architectures.  NSEAM 
has  been  designed  specifically  for  these  types  of  computer  architectures  while  yielding  the  same  high- 
order  accuracy  as  NOGAPS. 
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TRANSITIONS 


Improved  algorithms  for  model  processes  will  be  transitioned  to  6.4  (PE  0603207N)  as  they  are  ready, 
and  will  ultimately  be  transitioned  to  FNMOC  with  future  NOGAPS  upgrades. 


Figure  4:  Scalability  of  the  NRL ’s  spectral  element  atmospheric  model  (NSEAM)  compared  to 
NOGAPS  for  the  operational  resolution  T239L30  on  an  IBM  SP4.  The  number  of  simulation  days 
per  wall  clock  hours  is  given  as  a  function  of processor  number. 

RELATED  PROJECTS 

Some  of  the  technology  developed  for  this  project  will  be  used  immediately  to  improve  the  current 
spectral  transform  formulation  of  NOGAPS  (NRL  BE-35-2-18). 
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